On the Evaluation of Marginal Improvements in Pronunciation Variation Modeling for Spanish

In the context of large vocabulary speech recognition systems it is of major importance to accurately model the allophonic variations to be faced in a real world task. Evaluation of which variants are actually improving the system performance is crucial, as it determines the acceptance of the pronunciation alternatives used. Traditional approaches in this direction use different criteria and, typically, evaluation only cares about the global impact of the augmented dictionaries in the error rate, so that this lead to little further insight on till what extent the proposed variations are actually working or not. Our proposal in this paper is also evaluating the marginal improvement due to every pronunciation variation used (initially restricted to rule-based variant generation), defining specific improvement metrics. We experimentally show how these metrics actually show the improvement achieved by the application of rules when dealing with certain pronunciations (or speakers in general) while their global impact in error rate may not be statistically significant.